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INTRODUCTION 

Controlled  Flight  Into  Terrain  (CFIT)  accidents  continue  to  be  a  major  source  of  loss  of  aircraft  and 
aircrew  in  both  military  and  civil  aviation  (Cooper,  1995;  Miller,  Price,  and  Darrah,  1987;  Proctor,  1997). 
These  accidents  usually  occur  in  reduced-visibility  conditions,  and  with  no  prior  awareness  of  impending 
collision  on  the  part  of  the  flight  crew  (Scott,  1996).  While  warning  and  automated  collision  avoidance 
systems  have  reduced  the  rate  of  CFIT  accidents,  these  systems  focus  on  symptoms  rather  than  underlying 
causes.  Often,  a  pilot’s  first  response  to  a  Ground  Proximity  Warning  System  (GPWS)  alert  is  an  attempt 
to  verify  the  accuracy  of  the  alert  rather  than  taking  immediate  corrective  action  (Corwin,  1995).  Data  from 
the  Air  Force  Safety  Center  indicate  that  CFIT  accidents  occur  in  a  wide  variety  of  missions  and  aircraft. 
In  the  last  ten  years,  CFIT  accidents  have  cost  the  Air  Force  98  aircraft,  190  lives,  and  $1.68  billion 
(Moroze  and  Snow,  1999).  Also,  despite  inclusion  of  GPWS,  losses  due  to  CFIT  accidents  show  little 
sign  of  decreasing  (see  Figure  1). 
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Figure  1.  Controlled  flight  into  terrain  in  the  US  Air  Force,  FY87  through  FY96. 


While  the  Enhanced  Ground  Proximity  Warning  System  (EGPWS)  now  being  purchased  by  some 
airlines  is  an  improvement  on  GPWS,  it  is  still  primarily  a  warning  system  -  and  one  that  requires 
diversion  of  attention  from  primary  flight  references.  From  an  Air  Force  perspective,  both  GPWS  and 
EGPWS  are  less  than  optimal  because  they  are  emissive  systems  -  the  radar  altimeter  associated  with  these 
systems  can  serve  as  a  beacon  for  surface-to-air-missiles  (SAMs)  and  anti-aircraft  artillery  (AAA),  and  can 
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also  be  jammed  or  spoofed.  The  same  can  be  said  of  terrain-following  radar  (TFR)  systems  and 
millimeter- wave  radar  (MWR)  systems. 

What  is  needed  in  addition  to  these  systems  is  a  non-emissive  system  that  improves  pilot  situation 
awareness  with  respect  to  spatial  orientation  and  terrain  in  and  near  the  flight  path,  and  does  not  require  the 
pilot’s  gaze  or  attention  to  be  diverted  from  possible  external  events,  the  tactical  situation,  and  his/her 
primary  flight  reference.  Such  a  system  would  promise  significant  progress  toward  two  important  USAF 
goals:  elimination  of  CFIT  and  ability  to  use  same  tactics  day  or  night  (i.e.,  regardless  of  visibility). 
USAF  operational  commands  would,  ideally,  like  to  be  able  to  fly  any  mission  at  night  and/or  in  adverse 
weather.  The  primary  objective  of  the  study  conducted  was  to  determine  the  extent  to  which  these  goals 
could  be  met  by  superimposing  a  flight  path  on  synthetic  terrain  imagery  (STI)  in  a  HUD  for  pilots 
performing  low-altitude  maneuvering  and  air-to-ground  weapons  delivery  in  such  reduced-visibility 
conditions.  Previous  research  (Reising,  Liggett,  Solz,  and  Hartsock,  1995)  has  shown  that  the  pathway 
HUD  used  in  this  study  (see  Figure  2)  improves  pilot  performance  with  respect  to  maintenance  of 
commanded  airspeed,  altitude,  and  heading  during  instrument  landings,  roughly  halving  RMS  error  when 
compared  to  a  traditional  HUD.  Further,  these  benefits  have  been  found  regardless  of  external  visibility 
condition  (Reising,  Liggett,  Kustra,  Snow,  Hartsock,  and  Barry,  1998).  Unfortunately,  other  research  has 
demonstrated  that,  while  ego-centered  flight  path  displays  improve  performance,  they  also  reduce  global 
situation  awareness  (Olmos,  Liang,  and  Wickens,  1997).  It  was  hoped  that  adding  synthetic  terrain 
(graphical  representation  of  a  digital  terrain  elevation  database)  to  the  pathway  HUD  used  in  this  study 
would  offset  this  deficit,  at  least  with  respect  to  surrounding  terrain. 


FLIGHT  PATH  ATTITUDE 


Figure  2.  Basic  format  of  pathway  HUD  symbology  used. 

A  secondary  goal  of  this  research  was  to  determine  optimal  parameters  for  the  underlying  synthetic 
terrain  display  in  a  HUD.  Superposition  of  the  pathway  upon  STI  is  designed  to  provide  better  motion 
parallax,  compression,  splay,  and  edge  discontinuity  cues  than  STI  alone  (Dyre,  1997;  Wickens,  1992). 
Research  on  inclusion  of  synthetic  terrain  and  digital  terrain  elevation  data  in  the  cockpit  has  so  far  focused 
on  head-mounted  displays  (HMDs)  (Corwin,  Whillock,  and  Groat,  1994;  Rate,  Probert,  Wright,  Corwin, 
and  Royer,  1994),  head-down  displays  (Williams  and  Mitchell,  1993),  or,  in  many  cases,  terrain-following 
and  navigation  capabilities  without  a  terrain  display  (Hewitt,  1995).  An  HMD  would  seem  an  obvious 
choice  for  a  combination  pathway/STI  display  because  of  the  infinite  field  of  regard  it  gives  the  pilot  of  the 
STL  However,  HMDs  are  not  yet  as  proven  a  technology  in  high  performance  fighter  aircraft  as  the 
venerable  HUD,  and  there  are  some  implementation  problems  or  issues  specific  to  HMDs  that  can  be 
avoided  or  alleviated  through  use  of  a  HUD.  These  include  optical  distortions,  helmet  slippage,  other 
static  and  dynamic  misregistration  problems  (Azuma,  1997),  and  any  possible  adverse  side  effects  on  the 
pilot  of  HMD  usage  (Draper,  Prothero,  and  Virre,  1997;  Kennedy,  Jones,  Lilienthal,  and  Harm,  1994; 
Mon-Williams,  Warm,  and  Rushton,  1996;  Stanney,  Kennedy,  and  Drexler,  1997). 


While  some  data  exist  showing  that  pilots  prefer  synthetic  terrain  to  a  LANTIRN  (Low  Altitude 
Navigation  and  Targeting  Infrared  Night)  system  display  when  performing  simulated  low-altitude 
maneuvers  at  night,  data  regarding  pilot  performance  are  scarce.  The  data  provided  by  (Corwin  et  ah, 
1994;  Rate  et  al.,  1994)  indicate  that,  in  displaying  synthetic  terrain,  there  is  a  need  to  provide  as  much 
information  as  possible  with  respect  to  perspective,  apparent  size,  and  optical  flow  while  minimizing 
display  clutter.  These  authors  found  the  synthetic  terrain  imagery  (STI)  format  most  preferred  by  pilots  to 
be  a  full  grid  or  mesh.  This  format  was  included  as  a  condition  in  the  current  study  (see  Figure  3).  The 
partial  grid  and  texture-map  conditions  included  in  the  current  research  represent  attempts  to  improve  on 
this  format  by  providing  more  and  better  cues  to  accurate  perception  of  self-motion  and  depth. 
Specifically,  the  partial  grid  format  (Figure  4)  is  designed  to  provide  the  same  linear  perspective  (or  splay), 
compression,  edge  discontinuity,  and  optical  flow  information  as  the  grid  format,  but  with  fewer  pixels  to 
occlude  the  pilot’s  view  of  the  outside  world.  Also,  it  is  designed  to  facilitate  navigation  awareness  by 
providing  some  orientation-specific  information.  The  texture-map  condition  (Figure  5)  represents  a  format 
with  greater  optical  flow  and  it  includes  texture  gradient  as  a  depth  cue:  but  edge  discontinuity  and  linear 
perspective  are  minimal  and  it  probably  represents  the  most  clutter  of  the  three  formats  tested.  All  formats, 
by  including  the  pathway  and  its  associated  follow-me  aircraft,  provide  more  depth  and  self-motion 
information  than  STI  alone  through  the  addition  of  apparent  size,  object  interposition,  and  motion  parallax 
cues.  The  pathway  symbology  all  by  itself  (Figure  6)  served  as  a  baseline  for  the  STI  conditions.  The 
utility  of  these  formats  in  varying  visibility  conditions  is  also  a  concern.  It  seems  possible  that  an 
STI/Pathway  combination  that  does  well  in  darkness  might  not  be  the  best  in  IMC  (Instrument 
Meteorological  Condition)  day  conditions,  especially  partial  IMC  (e.g.,  in  and  out  of  clouds).  We  assume 
that  when  pilots  have  good  external  visibility,  a  synthetic  terrain  display  would  neither  be  needed  nor  used. 
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Figure  3.  Grid  Format  STI. 

Figure  5.  Texture-map  Format  STI. 

Figure  4.  Partial  Grid  Fonnat  STI.  Figure  6.  Pathway  symbology  without  HUD  STI. 


The  final  objective  of  this  research  was  to  test  two  measures  of  situation  awareness:  the  Situation 
Awareness  adaptation  of  the  Subjective  Workload  Dominance  (SA-SWORD)  technique  (Vidulich  and 
Hughes,  1991)  and  the  Situation  Awareness  Global  Assessment  Technique  (SAGAT)  (Endsley,  1995). 
SA-SWORD  is  a  subjective  measure,  using  a  paired-comparison  form  presented  shortly  after  testing  to 
elicit  relative  rankings  of  all  conditions  on  SA.  SAGAT  is  an  objective  measure  in  which  the  simulation 


is  briefly  halted  to  ask  pilots  SA-relevant  questions.  The  accuracy  with  which  pilots  answer  these 
questions  is  then  the  measure  of  SA.  Previous  research  (e.g.,  Boag,  Neale,  and  Neal,  1999)  has  found  that 
objective  and  subjective  measures  of  SA  can  differ,  presumably  because  pilots  essentially  “don’t  know 
what  they  don’t  know.”  We  wished  to  compare  and  contrast  SA-SWORD  and  SAGAT  to  determine  the 
possible  differences  between  objective  and  subjective  SA  measures  in  this  scenario,  and  to  gauge  the 
intrusiveness  (if  any)  and  effect  on  performance  of  the  simulation  halts  associated  with  the  use  of  SAGAT. 

In  summary,  building  on  previous  work,  there  were  three  primary  objectives  of  this  study:  1)  to 
determine  the  impact  on  situation  awareness  and  performance  of  four  configurations  of  a  combined  pathway 
and  synthetic  terrain  display  in  a  low-level  ingress  scenario,  2)  to  determine  the  interaction  of  these 
configurations  with  visibility  (if  any),  and,  3)  to  examine  two  situational  awareness  measurement 
techniques  for  their  utility  in  this  and  future  experiments. 

METHOD 

Subjects.  Twelve  pilots  participated  in  this  study.  Subject  pilots  were  required  to  have  some 
experience  in  a  HUD-equipped  aircraft  and  flying  military  fighter/attack  aircraft.  All  were  male  with 
military  flying  experience  ranging  from  1,300  to  3,775  hours,  with  an  average  of  2,634  hours. 

Apparatus.  This  study  was  conducted  in  a  single  seat  fighter  cockpit  known  as  the  Panoramic 
Cockpit  Control  And  Display  System  (PCCADS)  (Figure  7).  The  cockpit  has  a  15”  x  20”  monitor  that  can 
be  configured  through  software  to  display  any  number  of  separate  formats.  This  monitor  contains  a  full¬ 
screen  touch-sensitive  overlay  for  activating  displayed  switches.  A  50°  horizontal  x  40°  vertical  field  of 
view  (FOV)  forward  out-the-window  scene  was  projected  on  a  screen  in  front  of  the  simulator.  In  addition, 
a  30°  horizontal  x  20°  vertical  FOV  FIUD  was  simultaneously  projected  on  the  out-the-window  screen. 
This  is  the  same  HUD  FOV  as  a  LANTIRN -equipped  F-16.  A  side-mounted  stick  and  a  throttle  provided 
the  pilot  with  control  of  the  F-16  aeromodel,  as  well  as  hands-on-throttle-and-stick  (HOTAS)  functions. 
The  cockpit  also  includes  a  sound  system  to  generate  engine  noise  and  auditory  events. 

The  head-down  display  was  configured  to  provide  two  8”  x  8”  and  two  4”  x  4”  multifunction  displays 
(MFDs).  A  moving  map  was  shown  on  the  bottom  left  MFD  consisting  of  a  top-down  view  of  the  Digital 
Terrain  Elevation  Data  (DTED)  database,  with  terrain  at  or  above  the  pilot’s  current  altitude  color-coded 
red.  The  pilot’s  commanded  route,  waypoints,  target,  and  ownship  were  also  represented  on  this  map.  An 
instrument  suite  showing  airspeed,  attitude  director  indicator,  altimeter,  and  compass  with  horizontal 
situation  indicator  was  displayed  on  the  bottom  right  MFD.  The  top  left  MFD  showed  a  Stores  Status 
Display  (SSD)  and  the  top  right  MFD  showed  a  Threat  Electronic  Warfare  System  (TEWS)  display.  The 
SSD  showed  bombs  aboard  the  aircraft  and  when  released,  while  the  TEWS  showed  surface-to-air  missile 
(SAM)  radar  direction  and  distance  during  SAM  alerts.  These  displays  are  shown  in  Figure  8. 

Experimental  Design.  The  study  used  a  2  x  4  x  2  mixed-factors  design  with  two  within-subjects 
factors  and  one  between-subjects  factor.  The  first  within-subjects  factor  was  visibility  (IMC  Day  and  1MC 
Night).  The  second  within-subjects  factor  was  STI  type  (none,  grid,  partial  grid,  and  texture-map).  The 
between-subjects  factor  was  type  of  situation  awareness  measurement  (SA-SWORD  or  SA-SWORD  and 
SAGAT). 


Figure  7.  PCCADS  cockpit  simulator  used 
throughout  the  experiment. 


Figure  8.  Head-down  displays  common  to  all 
conditions. 


Independent  Variables.  The  two  out-the-window  visibility  conditions  were  IMC  Day  (visibility 
mile  with  a  100-foot  ceiling:  equivalent  to  ILS  CAT  II)  and  IMC  Night  (zero  visibility  with  zero  ceiling: 
equivalent  to  ILS  CAT  IIIc).  The  four  types  of  STI  were:  none,  grid,  partial  grid,  and  texture-map,  shown 
previously  in  Figures  3-6.  In  the  grid  condition,  points  in  the  DTED  database,  used  as  the  basis  for 
creating  the  STI,  were  connected  longitudinally  and  laterally  by  lines.  In  the  partial  grid  condition,  points 
in  the  DTED  database  were  displayed  as  Ts  (or  three  parts  of  a  cross).  Each  T  was  drawn  as  if  it  were 
laying  flat  upon  the  synthetic  terrain,  10M  long  x  20M  wide.  In  the  texture-map  condition,  a  terrain 
texture  map  was  overlaid  on  the  DTED  database.  The  hue  of  this  texture-map  was  the  same  as  that  of  the 
pathway,  but  with  varying  luminance.  In  all  STI  conditions,  the  maximum  luminance  of  the  synthetic 
terrain  was  half  that  of  the  pathway  symbology.  A  feature  of  the  STI  in  the  grid  and  partial  grid  conditions 
was  that,  when  pilots  descend  below  500  ft.  above  ground  level,  the  density  of  the  grid  tripled,  simulating 
a  shift  from  DTED  Level  I  (one  elevation  point  every  100  meters)  to  DTED  Level  II  (one  elevation  point 
every  thirty  meters).  For  half  the  subjects,  situation  awareness  was  measured  using  SAGAT  and  SA- 
SWORD;  for  the  other  half,  situation  awareness  was  measured  using  SA-SWORD  only. 

Dependent  Variables.  The  dependent  variables  for  on-path  flight  performance  were  root  mean  square 
(RMS)  error  in  lateral  deviation,  vertical  deviation  and  airspeed  deviation  from  the  commanded  path.  For 
purposes  of  analysis,  pilots  were  counted  as  off-path  when  the  aircraft  had  completely  left  the  commanded 
path  after  receiving  a  TEWS  alert.  The  dependent  variables  for  off-path  performance  were  Gs  pulled  during 
evasive  maneuvers,  ground  impacts,  and  time  to  leave  the  path  after  a  TEWS  alert  .  CEP  (circular  error 
probability)  was  recorded  as  a  measure  of  air-to-ground  weapons  delivery  performance.  SA-SWORD  and, 
for  half  the  subjects,  SAGAT  were  used  to  measure  situation  awareness.  A  questionnaire  was  administered 
to  solicit  pilot  biographical  data  and  subjective  opinions  regarding  the  display  combinations  and 
symbology  elements  tested. 

Procedure.  Pilots  received  training  in  both  the  classroom  and  in  the  cockpit.  Classroom  training 
included  briefing  the  subjects  on  the  purpose  of  the  study,  describing  the  different  experimental  conditions, 
and  explaining  how  the  TEWS  alerts  should  be  responded  to:  by  dodging  left  or  right  off  the  path  and 
“into  the  weeds”  (i.e.,  going  as  low  as  they  could  to  get  under  the  radar)  as  soon  as  possible.  Cockpit 
training  consisted  of  cockpit  familiarization  and  flying  a  practice  profile  to  demonstrate  the  pathway 
symbology  and  allow  the  pilot  to  become  familiar  with  the  aeromodel. 

Subjects  flew  eight  different  ingress  scenarios  (all  MK-84  deliveries).  Each  ingress  scenario  contained 
two  pop-up  SAM  threats,  one  located  near  the  beginning  of  the  scenario  and  one  located  near  the  end  of  the 
scenario.  These  pop-up  SAM  threats  were  designed  to  force  pilots  off  the  path  so  that  data  could  be 
collected  in  the  absence  of  the  pathway.  Pilots  continued  to  fly  for  fifteen  seconds  after  a  TEWS  alert 
sounded,  after  which  the  simulation  was  briefly  interrupted  and  then  restarted  with  the  pilot  back  on  the 
ingress  path.  Subjects  flew  a  short  practice  trial  in  each  experimental  condition  immediately  prior  to  flying 
the  ingress  scenario  in  that  condition. 

Pilots  in  the  SAGAT  plus  SA-SWORD  condition  answered  a  total  of  eighteen  questions  during  each 
ingress  scenario,  half  (selected  at  random)  during  an  on-path  simulation  interruption  and  half  after  one  of 
the  off-path  simulation  interruptions  that  followed  each  SAM  alert.  Subject  to  these  restrictions, 
interruptions  were  placed  at  random  within  the  ingress  scenario.  The  eighteen  questions  asked  of  each  pilot 
in  the  SAGAT  plus  SA-SWORD  condition  and  their  associated  dependent  measures  are  shown  in  Table  1. 


Table  1.  SAGAT  questions  used  in  the  study. 


SAGAT  Question 

Dependent  Measure 

1 .  Estimate  your  pitch. 

2.  Estimate  your  altitude  above  ground  level. 

Error  in  feet 

3 .  Estimate  your  current  bank  angle. 

Error  in  degrees 

4.  What  is  the  next  waypoint  number? 

Number  of  waypoints  off 

5 .  What  is  your  current  heading? 

Error  in  degrees 

6.  Does  your  flightpath  marker  currently  overlap  sky  or  terrain? 

Correct  /  Incorrect 

7.  What  direction,  using  the  hours  of  a  clock,  is  the  nearest  terrain  in  front 
of  you  that  is  above  your  current  altitude? 

Number  of  increments  off 

8.  Estimate  how  much  you  are  currently  above  or  below  your  commanded 
altitude. 

Error  in  feet 

9.  Estimate  how  much  you  are  currently  above  or  below  your  commanded 
airspeed. 

Error  in  knots 

10.  How  far  is  it  to  the  next  waypoint  (or  target  if  the  last  waypoint  has  been 
passed)? 

Error  in  miles 

11.  On-Path:  From  your  current  position,  what  is  the  safest  route  if  forced 
off  path  by  a  SAM  at  12  o’clock? 

Correct  /  Incorrect 

1 1 .  Off-Path:  From  your  current  position,  what  is  the  best  route  back  to 
your  commanded  path? 

Correct  /  Incorrect 

12.  From  your  current  position,  what  is  the  best  escape  route  to  avoid 
terrain? 

Correct  /  Incorrect 

13.  Give  a  detailed  description  of  the  terrain  at  10  o’clock,  halfway  to  the 
horizon. 

Correct  /  Incorrect 

14.  Give  a  detailed  description  of  the  terrain  at  12  o’clock,  halfway  to  the 
horizon. 

Correct  /  Incorrect 

15.  Give  a  detailed  description  of  the  terrain  at  2  o’clock,  halfway  to  the 
horizon. 

Correct  /  Incorrect 

16.  Give  a  detailed  description  of  the  terrain  at  10  o’clock,  at  the  horizon. 

Correct  /  Incorrect 

17.  Give  a  detailed  description  of  the  terrain  at  12  o’clock,  at  the  horizon. 

Correct  /  Incorrect 

18.  Give  a  detailed  description  of  the  terrain  at  2  o’clock,  at  the  horizon. 

Correct  /  Incorrect 

Presentation  of  the  ingress  scenarios  and  the  eight  combinations  of  weather  and  STI  were  counterbalanced 
using  a  standard  Latin  square.  Odd-numbered  subjects  were  assigned  to  the  SA-SWORD  only  condition, 
while  even-numbered  subjects  were  assigned  to  the  SAGAT  and  SA-SWORD  condition.  The  end  point  for 
each  ingress  scenario  was  determined  by  timing  the  path  during  the  escape  maneuver  for  ten  seconds  after 
weapon  release.  If  the  subject  was  unable  to  bomb  the  target,  the  task  was  counted  as  complete  when  the 
subject  executed  the  escape  maneuver  at  the  end  of  the  weapon  delivery  profile. 

RESULTS 

A  MANOVA  conducted  on  the  three  primary  flight  performance  measures  (RMS  error  in  airspeed, 
lateral  deviation,  and  vertical  deviation)  revealed  no  statistically  significant  effects  (_  <  0.05)  for  any  of  the 
independent  variables  manipulated  or  their  interactions.  Similarly,  no  statistically  significant  effects  were 
found  for  Gs  pulled  off-path,  MK-84  delivery  error,  or  reaction  time  to  SAM  alerts.  However,  with  respect 
to  SA-SWORD,  significant  main  effects  were  found  for  both  synthetic  terrain  format  (  F( 3,  30)  =  26.43,  p 
<  0.001)  and  visibility  (  F( 1,  10)  =  5.61  ,p  <  0.05).  These  are  illustrated  in  Figure  9.  Individually,  the 
only  SAGAT  question  for  which  a  significant  effect  was  found  was  Question  7  (see  Table  1).  The  effect  of 
synthetic  terrain  on  error  in  answering  this  question  was  significant  (  F(3,  15)  =  4.13,  p  <  0.05)  as  shown 
in  Figure  10. 
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Figure  9.  Effect  of  synthetic  terrain  format  and 
visibility  on  SA-SWORD  ratings  (error  bars  are 
90%  confidence  intervals). 
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Figure  10.  Effect  of  synthetic  terrain  format  on  error 
in  answering  SAGAT  Question  7  (error  bars  are  90% 
confidence  intervals). 


To  better  understand  the  impact  of  synthetic  terrain  and  visibility  on-  and  off-path,  SAGAT  data  were 
integrated  across  several  questions  into  six  categories  of  interest: 


1.  Level  I  Questions:  Questions  1-6  and  13-18  concerning  simple  awareness  of  task-relevant  events  and 
information. 

2.  Level  II  Questions:  Questions  7-10  concerning  awareness  of  and  understanding  of  the  meaning  of  task¬ 
relevant  events  and  information. 

3.  Level  III  Questions:  Questions  11  and  12  concerning  awareness  of  and  ability  to  forecast  future 
significant  task-relevant  events  and  information. 

4.  Piloting  Questions:  Questions  1-5  and  8-1 1  concerned  primarily  with  flying  the  aircraft. 

5.  Terrain  Questions:  Questions  6-7  and  12-18  concerned  primarily  with  terrain  near  the  flight  path. 

6.  Overall  Score:  Answers  to  all  SAGAT  questions  were  normalized  (i.e.,  smallest  error  set  to  zero  and 
largest  error  set  to  one)  and  the  mean  of  these  normalized  values  calculated  for  each  experimental 
condition. 


In  analyzing  these  data,  the  only  effects  found  to  be  significant  with  _  <  0.05  were  those  of  being  on-  or 
off-path  on  error  in  answering  Level  II  Questions  (  F(  1 ,  5)  =  10.38,  p  <  0.05)  and  Piloting  Questions  ( 
F(  1,  5)  =  23.51,  p  <  0.01).  These  effects  are  shown  in  Figures  11  and  12,  respectively.  Although  not 
significant  with  _  <  0.05,  for  purposes  of  illustration  and  comparison  with  SA-SWORD  results,  the  effect 
of  synthetic  terrain  (  F( 3,  15)  =  1.9,  p  =  0.18)  and  visibility  (  F(  1,  5)  =  4.92,  p  =  0.08)  on  overall 
SAGAT  scores  are  shown  in  Figure  13.  Finally,  the  impact  of  being  on-path  vs.  off-path  on  error  in 
answering  SAGAT  terrain  questions  is  shown  in  Figure  14.  Although  this  effect  was  not  statistically 
significant,  this  figure  is  included  to  provide  insight  into  the  question  of  whether  or  not  an  ego-centered 
path  display  decreases  global  SA  (e.g.,  awareness  of  surrounding  terrain).  As  shown  in  this  figure,  pilots 
missed  an  average  of  one  terrain  question  regardless  of  whether  they  were  asked  on-path  or  off-path. 
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Figure  11.  Effect  of  being  on-  and  off-path  on  error 
in  answering  Level  II  SAGAT  Questions  (error  bars 
are  90%  confidence  intervals). 
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Figure  13.  Effect  of  synthetic  terrain  and  visibility 
on  overall  error  scores  in  answering  SAGAT 
Questions  (error  bars  are  90%  confidence  intervals). 


Figure  12.  Effect  of  being  on-  and  off-path  on  error 
in  answering  Piloting  SAGAT  Questions  (error  bars 
are  90%  confidence  intervals). 
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Figure  14.  Effect  of  being  on-  and  off-path  on  error 
in  answering  Terrain  SAGAT  Questions  (error  bars 
are  90%  confidence  intervals). 


DISCUSSION 

The  results  of  this  study  suggest  that  inclusion  of  synthetic  terrain  in  a  F1UD  can  substantially 
improve  pilot  situation  awareness,  but  that  this  does  not  result  in  a  significant  benefit  to  flight  performance 
when  the  pilot  already  has  an  ego-centered  pathway  display  available  in  the  same  F1UD.  However,  two 
caveats  should  be  stated  in  accepting  this  conclusion: 

1.  These  results  apply  only  to  a  simulated  low-level  ingress  scenario.  Generalization  to  other 
applications  of  pathway-in-the-sky  displays  -  especially  precision  approaches  to  landing  -  is 
tenuous  and  should  await  further  research. 

2.  While  flight  performance  as  measured  by  RMS  error  in  airspeed,  lateral,  and  vertical  deviations 
was  not  significantly  affected  by  the  variables  manipulated  in  this  study,  there  were  four  ground 
impacts  during  the  ninety-six  ingress  scenarios  for  which  data  were  collected.  All  of  these  were  in 
the  IMC  Night.  Of  the  four,  three  were  in  conditions  without  head-up  synthetic  terrain  and  one 
was  in  the  partial  grid  condition.  Although  the  small  number  of  ground  impacts  prevents 


meaningful  statistical  analysis,  it  interesting  to  note  that  the  pattern  of  conditions  in  which  the 
impacts  occurred  closely  follows  the  pattern  of  SA  results  measured  in  this  experiment. 

Overall,  the  synthetic  terrain  format  yielding  the  best  SA  was  the  grid  format  as  measured  both 
subjectively  (SA-SWORD)  and  objectively  (SAGAT),  although  the  differences  between  this  condition  and 
the  texture  map  condition  were  marginal.  This  conclusion  was  supported  by  pilot  feedback  on  the 
subjective  questionnaire,  in  which  most  pilots  indicated  that  they  preferred  the  grid  format  overall 
(interestingly,  one  pilot  expressed  a  desire  to  be  able  to  switch  between  grid  and  texture  map  formats). 
Interactions  between  synthetic  terrain  format  and  visibility  were  not  found,  but  it  should  be  noted  that  a 
primary  visibility  condition  often  leading  to  CFIT,  transition  into  1MC,  was  not  tested. 

The  results  of  this  study  do  not  suggest  a  decrease  in  global  SA  associated  with  flying  an  egocentric 
pathway  display.  However,  unlike  other  studies  in  this  area  (e.g.,  Olmos  et  al.,  1997),  the  pathway  in  our 
study  was  displayed  on  a  HUD  (albeit  simulated)  rather  than  a  head-down  display  and  no  direct  comparison 
was  made  to  an  exocentric  or  perspective  frame  of  reference  display.  Given  our  findings  and  the  literature 
to  date,  it  seems  likely  that  any  decreases  in  global  SA  associated  with  the  use  of  these  display  formats 
may  be  more  dependent  on  frame  of  reference  (egocentric  vs.  exocentric)  and  field  of  view  of  the  display 
than  on  the  use  of  a  pathway-in-the-sky. 

Finally,  the  SA  metrics  tested  in  this  study  both  yielded  a  very  similar  pattern  of  results.  Differences 
found  with  SAGAT  were  not  as  strong  as  those  found  with  SA-SWORD,  but  because  of  the  experimental 
design,  the  SA-SWORD  data  were  based  on  twice  as  many  subjects  as  the  SAGAT  data.  No  results  were 
found  to  indicate  that  interruption  of  the  simulation  associated  with  SAGAT  affected  pilot  performance. 
Further,  even  when  asked  about  the  intrusiveness  of  this  technique  on  the  subjective  questionnaire,  pilots 
did  not  rate  it  as  intrusive.  These  results  support  similar  findings  by  others  (e.g.,  Endsley,  1995).  It  was 
our  experience  that  this  technique  is  associated  with  substantially  greater  overhead  than  SA-SWORD  in 
terms  of  simulation  preparation,  data  collection,  and  data  analysis.  However,  lack  of  intrusion  on 
performance,  the  face  validity  of  the  technique,  and  it’s  usefulness  as  a  diagnostic  tool  lead  us  to  conclude 
that  it  is  a  valuable  technique  and  one  that  we  will  use  in  the  future. 
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